Personalized sensitivity measurements and playback factors for adaptive and personalized media coding and delivery

ABSTRACT

A method for delivering media to a playback device including outputting first test media to be viewed by a first user. The method further includes receiving a first user input related to a first perception of the first test media by the first user and indicating a first personalized quality of experience of the first user with respect to the first test media. The method further includes generating a first personalized sensitivity profile including one or more viewing characteristics of the first user based on the first user input, and determining, based at least in part on the first personalized sensitivity profile, a first media parameter. The first media parameter is determined in order to increase an efficiency of media delivery to the first playback device over a network while preserving the first personalized quality of experience of the first user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 63/056,942, filed Jul. 27, 2020, and U.S. ProvisionalPatent Application No. 62/882,068, filed Aug. 2, 2019, both of which arehereby incorporated by reference in their entireties.

FIELD OF THE INVENTION

This application relates generally to delivery of visual media over anetwork to user devices for display of the visual media by the userdevices for viewing by a user.

SUMMARY

Various aspects of the present disclosure relate to devices, systems,and methods to provide personalized and adaptive media coding anddelivery based on playback-side information that is often collectedwithout individual sensors.

In one aspect of the present disclosure, there is provided a method fordelivering media to a playback device. The method may includeoutputting, with a first playback device and during a first testmeasurement session, first test media to be viewed by a first user. Themethod may further include receiving a first user input from the firstuser. The first user input may be related to a first perception of thefirst test media by the first user and may indicate a first personalizedquality of experience of the first user with respect to the first testmedia. The method may further include generating, with one or moreelectronic processors, a first personalized sensitivity profileincluding one or more viewing characteristics of the first user based onthe first user input. The method may further include determining, withthe one or more electronic processors and based at least in part on thefirst personalized sensitivity profile, a first media parameter. Thefirst media parameter may be determined in order to increase anefficiency of media delivery to the first playback device over a networkwhile preserving the first personalized quality of experience of thefirst user. The method may further include providing, over the network,first output media to the first playback device in accordance with thefirst media parameter. The first output media may be configured to beoutput with the first playback device.

In another aspect of the present disclosure, there is provided anelectronic computing device that may include a first playback deviceincluding a display. The display may be configured to output media to afirst user. The electronic computing device may also include one or moreelectronic processors communicatively coupled to the display. The one ormore electronic processors may be configured to output, with the firstplayback device and during a first test measurement session, first testmedia to be viewed by the first user. The one or more electronicprocessors may be further configured to receive a first user input fromthe first user. The first user input may be related to a firstperception of the first test media by the first user and may indicate afirst personalized quality of experience of the first user with respectto the first test media. The one or more electronic processors may befurther configured to generate a first personalized sensitivity profileincluding one or more viewing characteristics of the first user based onthe first user input. The one or more electronic processors may befurther configured to determine, based at least in part on the firstpersonalized sensitivity profile, a first media parameter. The firstmedia parameter may be determined in order to increase an efficiency ofmedia delivery to the first playback device over a network whilepreserving the first personalized quality of experience of the firstuser. The one or more electronic processors may be further configured toprovide, over the network, first output media to the first playbackdevice in accordance with the first media parameter. The first outputmedia may be configured to be output with the first playback device.

In another aspect of the present disclosure, there is provided a methodfor displaying a hybrid image on a playback device. The method mayinclude determining, with one or more electronic processors of anelectronic computing device, a first value of a media parametersupported by a media server and a network configured to stream media.The method may further include determining, with the one or moreelectronic processors, a second value of the media parameter supportedby the media server and the network. The method may further include atleast one of generating and selecting, with the one or more electronicprocessors, the hybrid image based on the first value of the mediaparameter and the second value of the media parameter such that thehybrid image includes a first interpretation corresponding to the firstvalue of the media parameter and a second interpretation correspondingto the second value of the media parameter. The method may furtherinclude displaying, on a display of the playback device, the hybridimage.

Other aspects of the embodiments will become apparent by considerationof the detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example adaptive bit rate (ABR) based media codingand delivery system according to embodiments described herein.

FIGS. 2 and 3 illustrate portions of adaptive bit rate (ABR) based mediacoding and delivery systems that are configured to determineindividualized/personalized viewing characteristics during testmeasurement sessions according to some embodiments described herein.

FIG. 4 illustrates a block diagram of a generic objective model beingtransformed into a personalized objective model via a model transformaccording to embodiments described herein.

FIG. 5 illustrates a graph including two different contrast sensitivityfunctions (CSFs) that show example relationships between contrastsensitivity and spatial frequency according to embodiments describedherein.

FIG. 6 illustrates a modified adaptive bit rate (ABR) based media codingand delivery system according to embodiments described herein.

FIG. 7 is a hardware block diagram of the playback system of FIG. 6according embodiments described herein.

FIG. 8 is a block diagram of the media server of FIG. 6 according toembodiments described herein.

FIG. 9 illustrates a flowchart of a method for delivering media to theplayback system of FIG. 6 according embodiments described herein.

FIGS. 10A-10C illustrate an example hybrid image in three differentsizes according to embodiments described herein.

FIGS. 11A and 11B illustrate graphs of example optimal ABR ladderestimates using hybrid images as test media in a multi-step binary treesearch during a test measurement session according to embodimentsdescribed herein.

FIGS. 12A and 12B illustrate example bandwidth of a network when usingan existing streaming method to stream media versus using the method ofFIG. 9 to stream media according to embodiments described herein.

FIGS. 13A and 13B illustrate another example of how the method of FIG. 9may allow more users/subscribers to stream media on a fixed-capacitynetwork without negatively impacting QoE according to embodimentsdescribed herein.

DETAILED DESCRIPTION

Visual media (e.g., images, videos, etc.) is deliverable via one or morecommunication networks to many different types of playbacksystems/devices (e.g., televisions, computers, tablets, smart phones,and the like) to be viewed by a user. In the visual media deliverychain, adaptive bit rate (ABR) streaming allows for improved networkresource management through adaptive selection of bit rate andresolution on a media ladder based on network conditions, playbackbuffer status, shared network capacity, and other factors influenced bythe network. Besides ABR streaming, other media delivery methods (whichalso may include coding methods or source coding methods) may similarlybe used to control one or more media parameters of an upstream videoencoder/transcoder/transrater such as bit rate, frame rate, resolution,etc.

However, up to this point, media delivery methods such as ABR streaminghave not taken into account additional factors to further improvenetwork resource management such as factors associated with playbacksystems/devices, with users' viewing capabilities, and with theenvironment in which the user is viewing the visual media. Rather, thesefactors are usually assumed to be ideal and uniform across differentusers/environments when performing content processing, coding, delivery,decoding and post-processing even though there exists a diverse range ofviewing conditions and variation in human visual performance thatgreatly affect a viewer's actual quality of experience (QoE).

For example, short-distance viewing can make users more sensitive indistinguishing between low- and high-resolution video content. Also,different viewers can have different visual sensitivity because offactors including but not limited to refractive error (even when wearingcorrective lenses), accumulation of floaters in the vitreous humor,age-related changes in color absorption by the lens, cataracts, ormacular degeneration. For example, contrast sensitivity of a user/viewermay decrease due to increased refractive error, increased disease,and/or increased age. Additionally, an individual's personal QoE canchange from place to place and from time to time, especially in mobileenvironments.

Detection of these visual sensitivity factors for each user/viewer canhelp estimate personalized QoE in real-world end-to-end systems andprovide opportunities for improving QoE and further improving mediadelivery efficiency. For example, a media delivery system can savebandwidth by transmitting a custom filtered version of the same video tomatch a user's/viewer's visual acuity or viewing distance from atelevision while maintaining personalized QoE of each user/viewer.

Several works have proposed to collect playback-side factors using avariety of sensors, aiming at selecting optimal bit rate and resolutionin media streaming, or feeding the information back to the mediapreprocessing, encoding and post-processing. However, the approach ofusing a variety of sensors to collect playback-side information isinsufficient and impractical for many playback systems (e.g.,television). It is insufficient because such sensors do not measure auser's innate visual acuity or sensitivity. It is impractical because itis overly burdensome to motivate and coordinate with televisionmanufacturers across the entire consumer display industry to equiptelevisions with the required sensors and metadata protocols. While thisburden is less for mobile devices which have many available sensorsalready, user privacy remains a concern, specifically when sensorscollect visual information about the users. Another issue with existingapproaches that utilize sensors to collect playback-side information isthat different models/brands of televisions have their own proprietaryupscaling and post-processing algorithms, and users may adjust varioustelevision settings such as brightness, contrast, or motion smoothing tosuit their preference.

To address the above-noted technical problems, the methods, devices, andsystems described herein include a new mechanism or protocol to shareparameters related to playback device characteristics and personalizedvisual-sensitivity factors with the upstream devices configured tocontrol the transmission of visual media to the playback devices. Themethods, devices, and systems described herein provide personalized andadaptive media delivery based on collected playback-side informationoften without using individual sensors. Additionally, the collectedplayback-side information may be indicative of personalized QoE fordifferent users and/or different viewing environments. The methods,devices, and systems described herein further improve network resourcemanagement/media delivery efficiency while maintaining personalized QoEfor each user.

FIG. 1 illustrates an example adaptive bit rate (ABR) based media codingand delivery system 100. The system 100 includes a media server 105 thatprovides media to a playback system 110 (i.e., playback device) over anetwork 115. Although FIG. 1 shows a single playback system 110, themedia server 105 may be configured to simultaneously stream the same ordifferent media to additional playback systems 110.

The playback system 110 may include many different types of playbacksystems such as a television, a tablet, a smart phone, a computer, andthe like. In some embodiments, the playback system 110 includes abuffer/decoder 120 and a playback renderer 125. The buffer/decoder 120may receive media from the server 105 over the network 115. Thebuffer/decoder 120 may buffer the received media and decode the receivedmedia to be output by the playback renderer 125. The buffer/decoder 120may include an electronic processor of the playback system 110 (e.g., amicroprocessor, a microcontroller, or other suitable processing device)as described in further detail below with respect to FIG. 7 . Theplayback renderer 125 may include an output device configured to displayimages and/or video. For example, the playback renderer 125 includes alight emitting diode (LED) display and/or a touch screen display asdescribed in further detail below with respect to FIG. 7 . The playbacksystem 110 is located in an environment 130. A user 135 is also locatedin the environment 130 and may view media that is output by the playbacksystem 110.

As illustrated in FIG. 1 , in some embodiments, the media server 105includes an ABR ladder 137 that is implemented by an electronicprocessor of the media server 105. In some embodiments, the media server105 receives one or more ABR requests 140 from the playback system 110to adjust a bit rate/quality decision of ABR streaming of media from themedia server 105 to the playback system 110 over the network 115. Forexample, the playback system 110 may retrieve and/or utilize a storedgeneric objective model 145 (e.g., device type, display resolution,GEO-based startup resolution, or more comprehensive models such as ITU-TP.1203, etc.) from a memory and use the generic objective model 145 tomonitor/measure streaming session and playback-related performanceinformation such as network connectivity metrics, media player bufferstatus, codec, bit rate, initial loading delay and stalling events, andthe like together with playback device 110 information such as displayresolution, screen size, playback device type, and the like. In someembodiments, packet header information and partial/complete bitstreamparsing may also be used to gather streaming session andplayback-related performance information. The streaming and playbackinformation is used to generate a generic quality of experience (QoE)estimation of the media streaming and playback of media. This QoEestimation may be used by the playback system 110 to influence the ABRrequest 140. For example, the playback system 110 periodicallydetermines each ABR request 140 for media segments based on a locallygenerated bandwidth estimate, buffer size, round-trip time, etc. with agoal of maintaining seamless playback. In other words, the genericobjective model 145 may be configured to allow the playback system 210to control media streaming based on at least one of resourceavailability of the network 115 and playback system parameters.

In some cases, playback systems 110 will simultaneously request two ormore segments representing the same time period in the media but encodedat different bit rates of the ABR ladder 137. Such a strategy may beinefficient and often leads to the playback system 110 requesting moredata than it needs for seamless playback. Such a strategy may also leadto the playback system 110 requesting a resolution/bit rate/frame ratecombination from the ABR ladder 137 that provides higher quality mediathat cannot be perceived by the user 135. In other words, existingABR-selection logic attempting to increase the delivered resolution/bitrate/frame rate beyond a sensitivity threshold of the user 135 does nottranslate to increased QoE for the user 135. In addition to nottranslating to increased QoE for the user 135, the requestedresolution/bit rate/frame rate combination may use more networkresources (e.g., more bandwidth) and/or may cost the user 135 additionalmoney (e.g., if the service provider of the media server 105 chargesbased on the amount of data provided to the playback system 110).

The above-noted problems of existing ABR-selection logic are caused bythe generic objective model 145 not taking into account personalized QoEwhen determining ABR requests 140. For example, the generic objectivemodel 145 may not take into account individualized/personalized viewingcharacteristics such as the lighting in the environment 130, a viewingdistance of the user 135 (i.e., the distance between the user 135 andthe playback renderer 125), vision sensitivities and capabilities of theeyes of the user 135 based on, for example, spatial frequency, and thelike. Rather, existing ABR-selection techniques assume that thesecharacteristics are the same for each environment 130 and for each user135 when, in fact, these characteristics may vary greatly betweenenvironments and/or users and impact the QoE of the user 135 to perceivemedia being displayed by the playback system 110.

While FIG. 1 and its corresponding explanation refer to the ABR ladder137 and ABR-selection techniques, ABR-related media delivery is merelyan example meant to represent general media delivery methods that may beimplemented by upstream devices such as the media server 105 and thenetwork 115. The ABR ladder 137 and ABR-selection techniques are alsoused as an example media delivery method throughout this applicationwith respect to additional figures (see, e.g., FIGS. 2, 3, and 11A-11B).However, the features disclosed herein may apply to any one of a numberof different media delivery methods (which also may include codingmethods or source coding methods) that are not based on the ABR ladder137 or ABR-selection techniques. In other words, the features describedherein may be used in conjunction with other media delivery methodsbesides ABR streaming that may similarly be used to control any mediaparameters of an upstream video encoder/transcoder/transrater such asbit rate, frame rate, resolution, etc. Additionally or alternatively,the features described herein may be used in conjunction with codingmethods or source coding methods that are used to code/process mediabefore the media is streamed. These coding methods and source codingmethods may be generally be referred to as media delivery methodsherein. In some embodiments, a media parameter includes a parameter thataffects media delivery from the media server 105 to the playback system210 (see FIGS. 2, 3 , and 6) over the network 115 (i.e., an upstreammedia parameter). In some embodiments, a media parameter additionally oralternatively includes a playback system parameter (i.e., a downstreamparameter) such as a brightness setting and/or a contrast setting of theplayback system 210.

FIGS. 2 and 3 illustrate portions of adaptive bit rate (ABR) based mediacoding and delivery systems 200 and 300 that are configured to determineindividualized/personalized viewing characteristics during testmeasurement sessions according to some embodiments. Test measurementsessions may be used to address the above-noted problems ofABR-selection logic (and/or other media delivery methods) byadditionally taking personalized viewing characteristics into accountwith (see FIG. 3 ) or without (see FIG. 2 ) using separate sensors tocollect this additional information.

FIG. 2 includes a playback system 210 that includes some similarcomponents as the playback system 110 of FIG. 1 . For example, theplayback system 210 includes the buffer/decoder 120 and the playbackrenderer 125. While not shown in FIG. 2 , the playback system 210 may becommunicatively coupled to a media server via a network similar to themedia server 105 and the network 115 shown in FIG. 1 . However, insteadof including the generic objective model 145 as shown in FIG. 1 , theplayback system 210 may generate a personalized (i.e., individualized)sensitivity profile (PSP) 215 for numerous different users 135 and/orenvironments 130. These personalized sensitivity profiles 215 may beused to provide ABR requests (or other requests with respect to othermedia delivery methods) to a media server streaming media to theplayback system 210. While the personalized sensitivity profiles 215 areexplained below as being generated by the playback system 210 (e.g., anelectronic processor of the playback system 210), in some embodiments,generation and storage of the personalized sensitivity profiles 215 mayadditionally or alternatively be performed by an electronic processor atthe media server, an electronic processor at a remote cloud computingcluster, or a combination thereof as described in further detail herein.

To generate a personalized sensitivity profile 215, the playback system210 implements a test measurement session where user responses 220 totest media 225 are collected from the user 135. During the testmeasurement session, the sensitivity of a user 135 is measured giventheir typical viewing conditions and environment. As an example, a user135 would sit in a typical viewing position (e.g., on a sofa in thefamily room, which represents a typical viewing condition in terms ofviewing distance, viewing angle, ambient luminance, and playback systemcharacteristics and settings). The playback system 210 then guides theuser 135 through a test measurement session to measure the audio-visualsensitivity of the user 135 in the environment 130 by followinginstructions provided by the playback system 210. In the session, theuser 135 may be asked to make one or more selections using a remotecontrol according to a series of images and/or videos presented by theplayback system 210. For example, the playback system 210 may displaymultiple images and request that the user select the image that appearsmost clear/in focus to the user 135. As another example, the playbacksystem 210 may display an image with multiple interpretations thatdepend on the vision capabilities of the user 135 and the viewingdistance of the user and request that the user select the interpretationthat is most dominant/evident to the user.

From the test measurement session, the playback system 210 and/or themedia server 105 may determine personalized viewing characteristics ofthe user 135 and/or the environment 130. For example, the user responses220 received during the test measurement session may indicate systemfactors such as playback system characteristics, playback parametersettings, post-processing algorithms of the playback system 210 (usuallyproprietary to the device manufacturer), and the like. As anotherexample, the user responses 220 received during the test measurementsession may indicate environmental factors such as viewing distance,viewing angle, ambient luminance, ambient noise, user expectation, andthe like. In some embodiments, user expectation refers to conscious orsubconscious psychological aspects of the user 135 that may affect theirperceived QoE. For example, the expectation level of the user 135 may behigher for media associated with a paid subscription than for othermedia such as free video on-demand services. As yet another example ofpersonalized viewing characteristics of the user 135 and/or theenvironment 130, the user responses 220 received during the testmeasurement session may indicate human factors such as sensoryacuity/vision sensitivities and capabilities of the user 135, age,gender, and the like. As indicated in FIG. 2 , in some embodiments, theplayback system 210 may additionally provide playback system information230 (e.g., device type, display resolution, etc.) in a similar manner asexplained above with respect to the system 100 of FIG. 1 . The playbacksystem 210 may use aggregated information 235 including the playbacksystem information 230 and the user responses 220 to the test media 225to determine the personalized sensitivity profile 215 for a particularuser 135 and/or environment 130.

As shown in FIG. 2 , the personalized sensitivity profile 215 mayinclude a user identification, other personal information (i.e., theindividualized viewing characteristics of the user 135 as determined bythe user responses 220), playback system information 230, and one ormore of geographic location, weather information, dates, and times atwhich the test media 225 was displayed to the user 135 during one ormore measurement sessions. In the example personalized sensitivityprofile 215 shown in FIG. 2 , each personalized sensitivity profile 215may be associated with a user and may include multiple sub-profiles fordifferent environments in which the user 135 has participated in testmeasurement sessions (e.g., different rooms of the user's home,different times of day, different weather information (e.g., sunnyversus cloudy), etc.). In other embodiments, each personalizedsensitivity profile 215 may be associated with a user and a specificenvironment such that each user may have multiple personalizedsensitivity profiles 215 that each correspond to different environmentsin which the user 135 has participated in test measurement sessions.

While the above-noted personalized viewing characteristics are notexplicitly collected by separate sensors (e.g., a sensor that measuresthe distance between the playback device 210 and the user 135), thesystem 200 is able to determine/estimate one or more of thesecharacteristics based on the user responses 220 to the test media 225during the test measurement session. Thus, in some embodiments,personalized viewing characteristic information is able to be gatheredfrom the user 135 and the environment 130 without the use of separate,explicit sensors. In other embodiments, separate explicit sensors may beused to provide additional information (see, e.g., FIG. 3 ).

FIG. 3 illustrates a portion of an adaptive bit rate (ABR) based mediacoding and delivery system 300 that is similar to the portion of thesystem 200 in FIG. 2 . However, the system 300 additionally includes oneor more environment sensors 305. For example, the playback system 210may include or may be communicatively coupled to a brightness/luminancesensor that measures the ambient light in the environment 130 (e.g.,integrated sensors, smart home sensors communicatively coupled to theplayback system 210, etc.). The environment sensors 305 may includeother smart home sensors located in the environment 130 that indicate,for example, whether lights are on/off, whether curtains coveringwindows are open/closed, etc. The environment sensors 305 may beconfigured to determine a time of day and a geographic location of theplayback system 210 for purposes of determining daylight hours, forexample. Environment sensing data 310 from the environment sensor(s) 305may be included in the aggregated information 235 and may be included inthe personalized sensitivity profile 215 as indicated in FIG. 3 . Insome embodiments, environment sensing data 310 included in theaggregated information 235 includes user provided information such aslevel of social activity in the environment 130 and/or viewing distance.

Collecting user responses 220 to test media 225 during a testmeasurement session implicitly takes into account many personalizedviewing characteristics that would otherwise be difficult, unrealistic,and/or obtrusive to explicitly collect using sensors. Additionally, insome situations, data explicitly collected using sensors may not allowfor an accurate determination of QoE of a the user 135. Accordingly, thesystems 200 and 300 provide a number of potential advantages andbenefits.

One example benefit relates to user variability. Two different users mayhave identical environmental characteristics (e.g., viewing distance,luminance, screen size, etc.). However, these two different users mayhave significantly different viewing capabilities due to, for example,differences in refractive error, age, and/or eye disease. Thus, usingphysical context/environmental characteristics alone to determine ABRrequests (or requests with respect to other media delivery methods) mayresult in different levels of QoE for these different users. In someembodiments, the personalized sensitivity profiles 215 of the systems200, 300 take these differences in personalized viewing capabilitiesinto account when determining ABR requests in order to prevent and/ordiminish reduction in personalized QoE for each user.

Another example benefit is the ease in which the systems 200 and 300 maybe implemented. In some embodiments, additional sensors need not beadded to playback systems 210. Along similar lines, user privacy isprotected as specific details of the user's environment may not beexplicitly measured and recorded in some embodiments. Rather, in someembodiments, a holistic evaluation of the user responses 220 that onlyimplicitly includes more detailed factors (such as the user's viewingdistance and viewing capabilities) is used to generate ABR requests tothe media server 105. In other words, the ABR request is based on acomposite measurement that is not based on the collection of independentattribute measurements from separate sensors. Rather, the ABR request inthe systems 200 and 300 is a holistic and implicit measurement of userQoE reflecting the combined effect of many factors, which are difficultor even impossible to collect explicitly.

Yet another example benefit relates to personalized content enhancementfor the user 135. In addition to enabling a more holistic and accurateestimation of end-user QoE, the systems 200 and 300 allow for enhancingthe media/content being played back to the user 135. Specifically, partsof a video frame that are either too small or too low in contrast to beperceived by the user (i.e., with spatial frequency or contrast beyond auser's measured contrast sensitivity function (CSF)) could be enhancedby the playback system 210. Examples of such enhancement may includecropping and magnifying the frame and/or applying local contrastadjustments to ensure that salient parts of the scene are visible to theuser 135 (i.e., within the user's measured CSF). Such enhancement mayimprove the user's viewing experience by helping the user 135 follow themedia/content and remain engaged with the media/content being watched.

Referring back to FIG. 1 , the generic objective model 145 to generatethe ABR request 140 may include any one of numerous models to estimateQoE. For example, one generic model is used to construct ABR-ladder 137(or another media delivery method), including multiple versions of bitrate/quality of media segments of the reference source media, typicallyby analyzing audio/video signals and optimizing coding efficiency.Another model is used for selecting appropriate bit rate/qualitydecision of ABR streaming based on network status and playback deviceloading. However, none of the example types of models take intoconsideration a user's personalized viewing characteristics. Thepersonalized sensitivity profiles 215 may be applied to any type ofgeneric model 145 to build a personalized objective model (POM) 405 froma generic objective model 145 (see FIGS. 4 and 6 ).

FIG. 4 illustrates a block diagram of a generic objective model 145being transformed into a personalized objective model 405 via a modeltransform 410. The transformation may occur according to one or moregoals desired to be achieved in media delivery control and management.For example, if the goal is to save media streaming bandwidth byselecting minimum bit rate in the ABR ladder 137 without degradingpersonalized QoE for each user 135, the systems 200, 300 may determinethe JND (just noticeable difference) of image/video in theresolution-bit rate grid space. The systems 200, 300 may then transformthe generic objective model 145 into the personalized objective model405 for each user 135/playback system 210 based on the personalized JNDof the user 135 as determined based on the personalized sensitivityprofile 215 of the user 135 in the environment 130 in which the user 135is using the playback system 210.

As another example, for a more sophisticated streaming managementencompassing real-time or non-real-time preprocessing, encoding,transcoding or transrating in the loop, the systems 200, 300 mayestimate personalized psychometric functions, such as spatial contrastsensitivity, temporal contrast sensitivity, and spatial-temporalcontrast sensitivity for achromatic and color-opponent stimuli toconstruct the personalized objective model 405. For example, FIG. 5illustrates a graph including two different contrast sensitivityfunctions (CSFs) that show example relationships between contrastsensitivity 505 and spatial frequency 510. The solid line curveillustrates an ideal CSF 515. The dashed line curve illustrates anexample user CSF 520 of the user 135 as determined based on the userresponses 220 to test media 225 during a given test measurement session.As shown in FIG. 5 , the user CSF 520 is translated and scaled comparedwith the ideal CSF 515 due to, for example, longer viewing distance inthe environment 130 than in an ideal viewing environment, lowerbrightness of the television screen than in an ideal viewingenvironment, and the user 135 having myopia/nearsightedness. Also asshown in FIG. 5 , compared to the ideal CSF 515, the user 135 has asmaller range of perceptible difference in contrast sensitivity 505relative to spatial frequency 510. In other words, the visioncapabilities of the user 135 are not as sensitive as those of an idealuser due to, for example, the environmental and personal conditionsmentioned above. For example, the user 135 may not be capable ofdistinguishing differences in contrast when the spatial frequency 510increases above a second value 530.

Accordingly, providing higher quality streaming of media that wouldallow for an ideal user with ideal contrast sensitivity in an idealenvironment to experience increased QoE would not actually result inincreased QoE for the example user 135 with the user CSF 520 shown inFIG. 5 . Thus, if providing this higher quality streaming comes at anexpense to the system 200, 300 (e.g., more bandwidth used because themedia is streamed used a higher bit rate), this expense experienced bythe system 200, 300 is essentially wasted because it does not result inimproved QoE for the user 135 viewing the streamed media.

To aid the system 200, 300 to control streaming of media from the mediaserver 105, the user CSF 520 is one example of data included in thepersonalized sensitivity profile 215 that is used to transform thegeneric objective model 145 into the personalized objective model 405(see FIG. 4 ). For example, the generic objective model 145 may use datacorresponding to the ideal CSF 515 or data corresponding to anothergeneric CSF that is not personalized to a user's viewing environment andpersonal vision capabilities. On the other hand, the personalizedobjective model 405 may use the user CSF 520 that has been scaled andtranslated to be personalized to the user's viewing environment andpersonal vision capabilities. The user CSF 520 may be used incombination with the other factors and/or algorithms included in thegeneric objective model 145 to create the personalized objective model405. As indicated by the above example of streaming higher quality mediathan can be perceived by the user 135, the personalized objective model405 may be utilized during video encoding, transcoding, and/ortransrating to improve coding efficiency to improve network efficiency(e.g., by reducing bandwidth) without affecting the personalized QoE ofthe user 135. For example, higher quality media may be streamed to moresensitive users 135 that are able to perceive the higher quality mediawhile lower quality media may be streamed to less sensitive users 135that are not able to perceive a difference in quality between the lowerquality media and the higher quality media.

Although the example graph shown in FIG. 5 refers to contrastsensitivity of the user 135, the generic objective model 145 may includevalues and/or functions related to other types of viewingcharacteristics that may be personalized based on the user responses 220to the test media 225 (e.g., temporal degradation related to video framerate, quantization degradation, etc.). For example, generic mediaparameter values and/or functions characterizing the generic temporaldegradation model may be replaced or retrained to generate personalizedvalues and/or functions in a personalized objective model 405.Additionally or alternatively, generic algorithms used to determinemedia parameters (i.e., streaming parameters), such as the ABR request140 (or requests with respect to other media delivery methods), may havecoefficients adjusted to generate a personalized objective model (POM)405.

FIG. 6 illustrates a modified adaptive bit rate (ABR) based media codingand delivery system 600. The system 600 is similar to the system 100 ofFIG. 1 but includes the personalized objective model (POM) 405 used togenerate the ABR request 140 (or a request with respect to another mediadelivery method) instead of the generic objective model 145. In someembodiments, the system 600 includes at least one of the portions of thesystems 200, 300 shown in FIGS. 2 and 3 . For example, as shown in FIG.6 , the personalized objective model 405 takes into account playbacksystem information 230, environment sensing information 310, andpersonalized viewing characteristic information from a PSP 215 asdetermined based on user responses 220 to test media 225 as describedabove.

Additional data sources that may be used by the POM 405 to generate theABR request 140 (or a request with respect to another media deliverymethod) include, but are not limited to real-time media player statusinformation including buffer size, playback status, player performancecharacteristics, etc. Another data source that may be used by the POM405 includes real-time network performance estimates such as throughputmeasured by the playback system 210, throughput measured from sensorslocated within the network 115, congestion notifications, latency,packet loss rate, etc. Another data source that may be used by the POM405 includes content metadata including bit rate, resolution, framerate, bit-depth per sample, chroma sampling, source coding method(including Level & Profile), color space, Supplemental EnhancementMessages (SEI), composition playlist(s), group of pictures (GOP) size,instantaneous decoding refresh (IDR) frame(s), maximum frame-averagelight level (MaxFALL), maximum content light level (MaxCLL),electro-optical transfer function (EOTF), language, service type, scenedescriptions (including boundary information), number of audio channels,audio sample rate, audio sample bit depth, audio service type, digitalsigning method, SCTE 35 messages, caption data, program loudness,regulatory information, ratings information, etc. In some embodiments,the additional data sources described herein may be referred to as mediaparameters.

Another data source that may be used by the POM 405 includes networkoperator policy parameters including maximum allowable bit rate, spatialresolution, frame rate, etc. per downstream and/or upstream channel orchannel equivalent. This example data source may allow for network-wideand cross-session optimizations. Another data source that may be used bythe POM 405 includes playback environmental sensor information 310 asexplained above (e.g., ambient luminance levels, ambient audio noiselevels, number of people viewing the streamed content, distance from thescreen of each viewer, etc.). Another data source that may be used bythe POM 405 includes ancillary mobile device information such asdistance from the primary playback system 210, mobile device sensorinformation, etc. Another data source that may be used by the POM 405includes real-time user/viewer preferences that may be entered by theuser 135 and stored by a memory of one of the devices included in thesystem 600.

FIG. 7 is a hardware block diagram of the playback system 210 (i.e.,playback device) according to one example embodiment. As mentionedabove, the playback system 210 may include many different types ofplayback systems such as a television, a tablet, a smart phone, acomputer, and the like. In the embodiment illustrated, the playbacksystem 210 includes a first electronic processor 705 (for example, amicroprocessor or other electronic device). The first electronicprocessor 705 includes input and output interfaces (not shown) and iselectrically coupled to a first memory 710, a first network interface715, an optional microphone 720, a speaker 725, and a display 730. Insome embodiments, the playback system 210 includes fewer or additionalcomponents in configurations different from that illustrated in FIG. 7 .For example, the playback system 210 may not include the microphone 720.As another example, the playback system 210 may include one or moreadditional input devices such as a computer mouse and/or a keyboard thatreceive inputs from a user of the playback system 210. As yet anotherexample, the playback system 210 may include environment sensors such asan ambient light sensor and/or a location tracking device (e.g., aglobal positioning system (GPS) receiver). In some embodiments, theplayback system 210 performs functionality other than the functionalitydescribed below.

The first memory 710 may include read only memory (ROM), random accessmemory (RAM), other non-transitory computer-readable media, or acombination thereof. The first electronic processor 705 is configured toreceive instructions and data from the first memory 710 and execute,among other things, the instructions. In particular, the firstelectronic processor 705 executes instructions stored in the firstmemory 710 to perform the methods described herein.

The first network interface 715 sends and receives data to and from themedia server 105 over the network 115. In some embodiments, the firstnetwork interface 715 includes one or more transceivers for wirelesslycommunicating with the media server 105 and/or the network 115.Alternatively or in addition, the first network interface 715 mayinclude a connector or port for receiving a wired connection to themedia server 105 and/or the network 115, such as an Ethernet cable. Thefirst electronic processor 705 may receive one or more data streams (forexample, a video stream, an audio stream, an image stream, and the like)over the network 115 through the first network interface 715. The firstelectronic processor 705 may output the one or more data streamsreceived from the media server 105 through the first network interface715 through the speaker 725, the display 730, or a combination thereof.Additionally, the first electronic processor 705 may communicate datagenerated by the playback system 210 back to the media server 105 overthe network 115 through the first network interface 715. For example,the first electronic processor 705 may determine and send the ABRrequest 140 mentioned previously herein to the media server 105. Themedia server 105 may then transmit one or more media streams to theplayback system 210 in accordance with the ABR request 140 from theplayback system 210.

The display 730 is configured to display images, video, text, and/ordata to the user 135. The display 730 may be a liquid crystal display(LCD) screen or an organic light emitting display (OLED) display screen.In some embodiments, a touch sensitive input interface may beincorporated into the display 730 as well, allowing the user 135 tointeract with content provided on the display 730. In some embodiments,the display 730 includes a projector or future-developed displaytechnologies. In some embodiments, the speaker 725 and the display 730are referred to as output devices that present media streams and otherinformation to a user 135 of the playback system 210. In someembodiments, the microphone 720, a computer mouse, and/or a keyboard ora touch-sensitive display are referred to as input devices that receiveinput from a user 135 of the playback system 210.

FIG. 8 is a block diagram of the media server 105 according to oneexample embodiment. In the example shown, the media server 105 includesa second electronic processor 805 electrically connected to a secondmemory 810 and a second network interface 815. These components aresimilar to the like-named components of the playback system 210explained above with respect to FIG. 7 and function in a similar manneras described above. In some embodiments, the second network interface815 sends and receives data to and from playback systems 210 via thenetwork 115. In some embodiments, the media server 105 includes fewer oradditional components in configurations different from that illustratedin FIG. 8 . For example, the media server 105 may additionally include adisplay such as a touch screen to allow a backend user to reprogramsettings or rules of the media server 105. In some embodiments, themedia server 105 performs functionality other than the functionalitydescribed below.

While FIGS. 7 and 8 show separate block diagrams of the playback system210 and the media server 105, in some embodiments, the media server 105,one or more playback systems 210, a remote cloud-computing cluster thatcommunicates over or forms a part of the network 115, or a combinationthereof is referred to an electronic computing device that performs thefunctionality described herein. For example, the electronic computingdevice may be a single electronic processor (for example, the secondelectronic processor 805 of the media server 105) or a plurality ofelectronic processors located in the media server 105. In otherembodiments, the electronic computing device includes multipleelectronic processors distributed across different devices. For example,the electronic computing device is implemented on one or more of thefirst electronic processors 705 of the playback systems 210, the secondelectronic processor 805 of the media server 105, and one or moreelectronic processors located in one or more other devices located at aremote location or at a remote cloud-computing cluster that communicatesover or forms a part of the network 115. In some embodiments, the remotecloud-computing cluster includes a Software-Defined-Network(SDN)/Network Function Virtualization (NFV)-enabled access-network.

In some embodiments, the device(s) that implements the POM 405 maydetermine the goal and function of the POM 405. For example,implementation of the POM 405 within the playback system 210 allows fordecentralized operation in the absence of network operator or othercontrol signals. On the other hand, implementation of the POM 405 withinthe media server 105 and/or the network 115 (e.g., as a networkvirtualized function (NVF) located on a software defined network (SDN)node) may simplify the deployment of network-wide QoE optimizations andother network operator policies (e.g., an optimization of networkservices to a desired subscriber QoE, Edge/Access-Network capacitytarget, or a combination of both).

One or more of the hardware components of the playback system 210 shownin FIG. 7 implements and/or makes up the functional components of theplayback system 210 shown in FIGS. 2, 3, and 6 . For example, the firstelectronic processor 705 (or multiple first electronic processors 705 ofthe playback system 210) may act as one or more of the buffer/decoder120 and the playback renderer 125. The first electronic processor 705may also determine the personalized sensitivity profile (PSP) 215, thepersonalized objective model (POM) 405, and the ABR request 140.

In some embodiments, one or more personalized sensitivity profiles(PSPs) 215 of one or more users and environments are stored in the firstmemory 710 of the playback system 210. The first memory 710 may storeadditional information such as general playback system information 230of the playback system 210 (e.g., screen size, product identificationnumber, and the like). In some embodiments, one or more personalizedsensitivity profile (PSPs) 215 of one or more users and environments areadditionally or alternatively stored in the second memory 810 of themedia server 105 and/or a memory of a remote cloud-computing clusterthat communicates over or forms a part of the network 115. In someembodiments, cloud-storage of user's PSP(s) 215 enables secure linkingto a user's wired/wireless Internet Service Provider (ISP) or networkdelivered media account (e.g. cable tv). Such linking may be useful fora network operator to leverage individual PSPs 215 for generating moreefficient media delivery across the portion(s) of their subscriber basewith a PSP 215 associated with their account as described in greaterdetail herein.

FIG. 9 illustrates a flowchart of a method 900 for delivering media to aplayback system 210 according to one example embodiment. The method 900is described as being performed by an electronic computing deviceincluding one or more of the electronic processors described previouslyherein. Although certain actions are explained as being performed by theelectronic processor(s) of a certain device, in other embodiments,electronic processors of other devices may perform the same actions.While a particular order of processing steps, message receptions, and/ormessage transmissions is indicated in FIG. 9 as an example, timing andordering of such steps, receptions, and transmissions may vary whereappropriate without negating the purpose and advantages of the examplesset forth in detail throughout the remainder of this disclosure.

At block 905, the media delivery method 900 is initiated. In someembodiments, the media delivery method 900 is initiated by the firstelectronic processor 705 of the playback system 210 in response to theuser 135 turning on the playback system 210 and/or requesting that adata stream be output by the playback system 210.

In response to the media delivery method 900 being initiated, at block910, one or more electronic processors of the electronic computingdevice retrieves stored personalized sensitivity profiles (PSPs) 215related to at least one of the user 135, the playback system 210, andthe environment 130 in which the playback system 210 is located. Forexample, the stored PSPs 215 may have been generated based on previoustest measurement sessions of the playback system 210. In someembodiments, a stored PSP 215 may be used by the electronic computingdevice to provide output media to the playback system 210 withoutperforming a new test measurement session. For example, when theplayback system 210 is being used in an environment 130 that has alreadyundergone a test measurement session by the same user 135, theelectronic computing device may use a stored PSP 215 corresponding tothe environment 130 and the user 135. In some embodiments, theelectronic computing device determines that the current environment anduser have already undergone a test measurement session by comparingplayback system information, environment information, and/or userinformation (e.g., user login information received by the playbacksystem 210) to stored information of PSPs 215. For example, theelectronic computing device may determine an identification number ofthe playback system 210, one or more characteristics of the environment130 (e.g., time of day, amount of ambient light, location of playbacksystem 210, and the like), and an identity of the user 135. If thisidentification information matches with a PSP 215 already stored in oneof the memories of the electronic computing device, the electroniccomputing device may control the providing of output media to and thedisplaying of output media by the playback system 210 according to thecorresponding previously-stored PSP 215 without proceeding to block 915to perform a new test measurement session.

On the other hand, the method 900 may continue to block 915 where theplayback system 210 is controlled to perform a new test measurementsession. Here, the new test measurement session can be a full lengthsession or a reduced length session based on a prior knowledges of oneor more stored PSPs 215. For example, if one or more of the above-notedtypes of identification information do not match with a stored PSP 215,the electronic computing device may perform a new test measurementsession and generate a new PSP 215 as explained in further detail below.In some embodiments, a new test measurement session is initiated by theuser 135 (e.g., via a user input on an input device of the playbacksystem 210). In some embodiments, the electronic computing device maydetermine that at least one characteristic of a previously-stored PSP215 has changed (e.g., a power interruption, a change in InternetProtocol (IP) address, a change in WiFi signal strength, anewly-detected peripheral device being coupled to the playback system210, a change in detected ambient light, a change in detected locationof the playback system 210, and the like). In response thereto, theelectronic computing device may instruct the playback system 210 torecommend that the user 135 participate in a new test measurementsession. For example, the playback system 210 may determine that theuser 135 is now viewing the playback system 210 at night rather thanduring the day (e.g., based on a time of day measurement, based on datareceived from environmental sensors 305, etc.). As another example, themedia server 105 may determine that a new playback system 210 that isnot associated with any previously-stored PSPs 215 has been connected tothe network 115. In response thereto, the media server 105 may transmita request to the playback system 210 that recommends the user 135participate in a test measurement session to generate a PSP 215.

Methods of implementing a test measurement session may includeintegration into the initial setup steps of a set-top box (STB), DigitalMedia Adapter (DMA), mobile device, or other playback system 210, as athird-party application running on playback system 210, or as a cloudservice that hosts both the test media 225 and/or the PSPs 215. As notedpreviously herein, test measurement session results may be saved locallyon the playback system 210 and/or remotely as part of a cloud serviceenabling cross-platform and cross-service compatibility.

At block 915, the electronic computing device outputs test media 225 tobe viewed by the user 135. The test media 225 may be generated by thefirst electronic processor 705 of the playback system 210 or may bereceived by the playback system 210 after being generated by the mediaserver 105. In some embodiments, the test media 225 is generated inorder to measure user sensitivity/quality of experience (QoE). Forexample, at block 920, the electronic computing device receives a userinput (i.e., user responses 220) from the user 135. The user input isrelated to a perception of the test media 225 by the user 135 andindicates a first personalized QoE of the user 135 with respect to thetest media 225.

In some embodiments, the electronic computing device determines the usersensitivity/QoE of the user 135 by generating an optotype acuitymeasurement using test media 225 including a Snellen chart or open ringpatterns. In some embodiments, the electronic computing deviceadditionally or alternatively determines the user sensitivity/QoE of theuser 135 by generating a contrast sensitivity function (CSF) measurementusing sine-wave gratings of different orientations (e.g., see FIG. 5 ).In some embodiments, the CSF measurement may include a quick CSF methodusing test media 225 including bandpass-filtered Sloan letters. In someembodiments, the electronic computing device additionally oralternatively determines the user sensitivity/QoE of the user 135 bydisplaying test media 225 in the form of an interactive game to beplayed by the user 135. In some embodiments, the electronic computingdevice additionally or alternatively determines the user sensitivity/QoEof the user 135 by taking user sensitivity measurements based on a setof image or video materials displayed as test media 225.

In some embodiments, the electronic computing device may display testmedia 225 in the form of hybrid images. In some embodiments, a hybridimage is static image that tends to have distinct interpretationsdepending on the user's viewing capabilities and environmental factors.As an example, human viewers lose their capability to see fine detailsof images as the viewing distance is increased, resulting in failing todistinguish between high- and low-resolution videos. In someembodiments, a hybrid image is a static image that produces two or moredistinct interpretations to a human user that change as a function ofspatial frequency range and/or viewing distance. Based on user responses220 to displayed hybrid images, the electronic computing device mayestimate dominant and non-dominant spatial frequency ranges of the user135 in the media viewing environment 130 without using an explicitsensor.

To create a hybrid image, two different source images may be processeddifferently to make a certain spatial frequency range dominant withrespect to each image. For example, a first source image may be low-passfiltered and a second source image may be high-pass filtered. Thelow-pass filtered source image may then be combined with (i.e.,overlayed on top of) the high-pass filtered source image to create ahybrid image. Because the sensitive region of a given image in spatialfrequency moves from lower frequencies to higher frequencies as theviewing distance of the user 135 is decreased, a human user more easilyperceives the high-pass filtered source image at shorter viewingdistances than at longer viewing distances. Conversely, a human usermore easily perceives the low-pass filtered source image at longerviewing distances than at shorter viewing distances. In other words,either the low-pass filtered source image or the high-pass filteredsource image is perceived by the user 135 as dominant depending on oneor more viewing characteristics of the user 135.

FIGS. 10A-10C illustrate an example hybrid image 1000 in three differentsizes with FIG. 10A showing the image 1000 in the largest size and FIG.10C showing the image 1000 in the smallest size. In the example shown, afirst source image including a face of a dog 1005 is low-pass filtered(see low-pass filtered source image 1050 of FIG. 10D) and combined witha second source image including a face of a cat 1010 that is high-passfiltered (see high-pass filtered source image 1060 of FIG. 10E). Asindicated by FIGS. 10A-10C, the face of the cat 1010 is easier toperceive for a human user in larger FIG. 10A than in smaller FIGS. 10Band 10C due the high-pass filtering of the second source image. In otherwords, due to high-pass filtering (see FIG. 10E), the hybrid image 1000includes only fine details of the second source image including the faceof the cat 1010 that are easier to perceive when the image 1000 islarger (i.e., a close/short viewing distance). Conversely, the face ofthe dog 1005 is easier to perceive for a human user in FIG. 10C than inFIGS. 10A and 10B. In other words, due to low-pass filtering (see FIG.10D), the hybrid image 1000 includes only coarse details of the firstsource image including the face of the dog 1005 that are easier toperceive when the image 1000 is smaller (i.e., a longer viewingdistance). To aid a viewer with visual interpretation of FIGS. 10A-10C,FIGS. 10A-10E have a nose 1015 of the cat 1010 labeled in each figureand a nose 1020 of the dog 1005 labeled in each figure.

Although the generation of hybrid images is explained above as includinglow-pass filtering and high-pass filtering different source images, insome embodiments, hybrid images are additionally or alternativelygenerated using different bandpass filters. In some embodiments, varyingthe size of source images results in scaling up or down in the spatialfrequency domain. Accordingly, in combination with filtering, varyingthe size of source images is another way to generate hybrid images.

By displaying a series/plurality of hybrid images as the test media 225during the test measurement session at block 915, the electroniccomputing device may be able to determine viewing characteristics of theuser 135 and environmental factors related to the playback system 210.In some embodiments, the electronic computing device may vary a size ofthe hybrid image displayed by the playback system 210. For example, theelectronic computing device may vary a size of the hybrid image until auser response 220 indicates that the user's perception of the hybridimage has changed from a first perception of the first source image to asecond perception of the second source image. Based on the size of thehybrid image being displayed at the time the user response 220 wasreceived and based on the resolution and screen size of the playbacksystem 210, the electronic computing device may be able to determine anestimated viewing distance of the user 135, an estimated CSF for theuser 135, and/or the like.

In some embodiments, the electronic computing device may vary cutofffrequencies of the low-pass filter and the high-pass filter (or of aband-pass filter) of each source image being used to create a hybridimage either randomly or adaptively based on previous user responses 220received during the test measurement session. For example, theelectronic computing device may receive a first user input related to afirst perception of a first hybrid image by the user. In responsethereto, the electronic computing device may generate a second hybridimage using filters where a cutoff frequency of at least one filter isbased on the first user input related to the first perception of thefirst hybrid image (e.g., see FIGS. 11A and 11B). The electroniccomputing device may then control the playback system 210 to output thesecond hybrid image to be viewed by the user 135.

In some embodiments, the electronic computing device may determine thecutoff frequencies of spatial filters (and/or another characteristicused to generate the hybrid image such as the size of the hybrid imagebeing displayed) based on playback system parameters and/or mediaparameters supported by the media server and the network 115. Forexample, the electronic computing device may determine the cutofffrequencies of spatial filters in conjunction with the available videoresolutions in the ABR ladder 137 of the media server 105 (or inconjunction with available values of other media parameters based onanother media delivery method being utilized by the media server 105).As another example, the electronic computing device may determine thecutoff frequencies based on available bit rates of the media server105/network 115, available frame rates of the media server 105/network115, device type of playback system 210, screen size of the display 730of the playback system 210, and/or other parameters/attributes mentionedpreviously herein.

In some embodiments, the electronic computing device determines a firstvalue of a media parameter supported by the media server 105 and thenetwork 115. The electronic computing device may also determine a secondvalue of the media parameter supported by the media server 105 and thenetwork 115. The electronic computing device may then at least one ofgenerate and select a hybrid image based on the first value of the mediaparameter and the second value of the media parameter such that thehybrid image includes a first interpretation corresponding to the firstvalue of the media parameter and a second interpretation correspondingto the second value of the media parameter (e.g., see FIGS. 11A and11B). The electronic computing device may then control the playbacksystem 210 to display the hybrid image on the display 730.

In some embodiments, the electronic computing device displays additionalhybrid images based on the user response(s) 220 to previously-displayedhybrid images as described previously herein. For example, theelectronic computing device may at least one of generate and select asecond hybrid image based on the first value of the media parameter anda third value of the media parameter (that is determined to be supportedby the media server 105 and the network 115) such that the second hybridimage includes a third interpretation corresponding to the third valueof the media parameter and a fourth interpretation corresponding to thefirst value of the media parameter.

In some embodiments, the hybrid images described in the above exampleare generated by the electronic computing device by overlaying sourceimages as described previously herein. In other embodiments, electroniccomputing device may retrieve previously-generated and stored hybridimages with characteristics corresponding to the values of the mediaparameter determined to be supported by the media server 105 and thenetwork 115.

During the test measurement session, the electronic computing device mayreceive, with an input device of the playback device, a user input fromthe user 135. The user input indicates that a first interpretation of ahybrid image is perceived by the user 135 when the hybrid image isdisplayed on the display 730. Based on the user input, the electroniccomputing device may determine that the user 135 is more sensitive tothe first value of the media parameter (e.g., a first spatial frequencyrange, viewing distance, resolution, and/or the like) than to the secondvalue of the media parameter (e.g., a second spatial frequency range,viewing distance, resolution, and/or the like). In some embodiments, theelectronic computing device generates a personalized sensitivity profile215 of viewing characteristics of the user 135 based on thedetermination that the user 135 is more sensitive to the first value ofthe media parameter. The personalized sensitivity profile 215 mayinclude the first value of the media parameter. In some embodiments, themedia server 105 may provide, over the network 115, output media to theplayback system of the user 135 in accordance with the personalizedsensitivity profile 215 as explained previously herein.

Continuing the immediately above example, the electronic computingdevice may determine, based on the user input, at least one of a subsetof spatial frequencies of the hybrid image (i.e., a contrast of thehybrid image) to which the user 135 is most sensitive and a sizing ofthe hybrid image to which the user 135 is most sensitive. In someembodiments, the viewing characteristics of the personalized sensitivityprofile 215 generated by the electronic computing device include the atleast one of the subset of spatial frequencies of the hybrid image towhich the user is most sensitive and the sizing of the hybrid image towhich the user is most sensitive.

As indicated by the above examples, use of hybrid images generated orselected based on media parameters and/or playback system parameters(i.e., media-centric parameters) during the test measurement session mayallow the electronic computing device to determine, for example, howdifferent media-centric parameters affect the user's personalized QoE.For example, the electronic computing device may determine how differentvideo resolutions of the ABR ladder 137 (or how different values of amedia parameter of another media delivery method) affect the user'spersonalized QoE. In other words, based on the user responses 220 to thetest media 225, the electronic computing device estimates a range ofdominant spatial frequencies influencing the user's perception andinvisible spatial frequencies. This perceptual information can be usedto improve the efficiency of media coding and delivery as explainedherein. For example, the lowest video resolution in the ABR ladder 137can be identified below which the user 135 starts to experience qualitydegradation compared with the full-resolution video.

FIGS. 11A and 11B illustrate graphs of example optimal ABR ladderestimates using hybrid images as the test media 225 in a multi-stepbinary tree search during a test measurement session. In someembodiments, the electronic computing device obtains available videoresolutions of media streaming of the media server 105 and the network115 (e.g., 360p, 540p, 720p, and 1080p) from a manifest file. Accordingto the video resolutions, the electronic computing device determinescutoff frequencies of the low-pass filter and the high-pass filter thatare to be applied to source images A and B to create hybrid images. Thevertical dotted lines shown in FIGS. 11A and 11B drawn along the rowsrepresent the upper frequency limit of the corresponding videoresolutions in the ABR ladder 137. For example, the spectral contents ofthe 540p video can take only up to the second vertical dotted line fromthe left. The top graph 1105, 1155 in each of FIGS. 11A and 11B shows apresumed contrast sensitivity function (CSF) as a function of spatialfrequency [cycle per pixel] for a user 135 at a certain viewingdistance. As shown in FIG. 11A, a peak 1110 of the sensitivity of theuser 135 lies between the 540p point and the 720p point.

The middle graph 1115 in FIG. 11A illustrates sensitivity of a firsthybrid image created from the filtered source images A and B. Asindicated by the graph 1115, the electronic computing device mayinitially set the spatial frequency at which a change in humanperception of the source images A and B of the first hybrid image mayoccur to be at 540p. Based on the user response 220 to the first hybridimage (i.e., test media 225), the electronic computing device determineswhich source image A or B is perceptually more dominant to the user 135.When the user response 220 indicates that source image B is moredominant, the electronic computing device may generate a second hybridimage represented by the bottom graph 1120 of FIG. 11A. As indicated bythe bottom graph 1120, the electronic computing device may set thespatial frequency at which a change in human perception of second sourceimages A and B of the second hybrid image may occur to be at 720p basedon the user response 220 selecting source image B from the display ofthe first hybrid image represented by the middle graph 1115. Throughdisplaying of multiple hybrid images that are dynamically/adaptivelyadjusted based on user responses 220, the electronic computing device isconfigured to narrow down the dominant frequency range perceptible tothe user 135. The electronic computing device may be configured todetermine an estimated CSF 1125 as shown in the top graph 1105 of FIG.11A based on the user responses 220 received throughout the testmeasurement session. Although only two iterations of graphsrepresentative of displayed hybrid images are shown in FIG. 11A, in someembodiments, the electronic computing device displays additional hybridimages (i.e., test media 225) and receives additional corresponding userresponses 220 during the test measurement session.

FIG. 11B is a similar example as FIG. 11A but corresponds to a differentuser 135, environment 130, and/or playback system 210 (e.g., the sameuser 135 and playback device 210 but a farther viewing distance thanthat of the example of FIG. 11A). The graphs 1155, 1160, and 1165generally correspond to the respective graphs 1105, 1115, and 1120 ofFIG. 11A with values adjusted according to the different viewingsituation as noted above. As indicated by the graph 1160, the electroniccomputing device may initially set the spatial frequency at which achange in human perception of the source images A and B of a firsthybrid image may occur to be at 540p. Based on the user response 220 tothe first hybrid image (i.e., test media 225), the electronic computingdevice determines which source image A or B is perceptually moredominant to the user 135. Unlike the example shown in FIG. 11A, when theuser response 220 indicates that source image A is more dominant, theelectronic computing device may generate a second hybrid imagerepresented by the bottom graph 1165 of FIG. 11B. As indicated by thebottom graph 1165, the electronic computing device may set the spatialfrequency at which a change in human perception of second source imagesA and B of the second hybrid image may occur to be at 360p based on theuser response 220 selecting source image A from the display of the firsthybrid image represented by the middle graph 1160. As explained abovewith respect to FIG. 11A, the electronic computing device may continuedisplaying hybrid images and receiving user responses 220 to determinean estimated CSF 1170 as shown in the top graph 1155 of FIG. 11B.

As shown in FIGS. 11A and 11B, the CSFs 1125 and 1170 are different thaneach other due to differences in one or more of users 135, environments130, and/or playback systems 210. For example, the CSF 1170 of FIG. 11Bhas a sensitivity peak 1175 at a lower resolution than the sensitivitypeak 1110 of the CSF 1125 of FIG. 11A. As another example, the overallrange of the CSF 1170 of FIG. 11B is less than that of the CSF 1125 ofFIG. 11A such that the user of FIG. 11A is able to distinguish betweendifferent resolutions greater than approximately 540p while the user ofFIG. 11B is not able to distinguish between different resolutionsgreater than approximately 540p.

As is evident from the above explanation, the CSFs 1125 and 1170 arepersonalized CSFs based on the user responses 220 received by theelectronic computing device in response to the displayed hybrid imagesand/or other test media 225. The personalized CSFs determined by theelectronic computing device are similar to the CSF 520 shown in FIG. 5and explained previously herein. In other words, instead of usinggeneric ABR logic (e.g., the ideal CSF 515 of FIG. 5 ) to control mediastreaming from the media server 105 to the playback system 210, theelectronic computing device may use personalized bit rate/resolutiondecisioning rules to do so. In some embodiments, the electroniccomputing device generates a new, personalized ABR ladder based on theuser responses 220 to the test media 225 during the test measurementsession.

In some embodiments, one or more stored PSPs 215 may influencecharacteristics of the test media 225 output by the playback system 210during the test measurement session. In some embodiments, the electroniccomputing device retrieves a previously-stored personalized sensitivityprofile (PSP) 215 and generates the test media 225 based on one or moreviewing characteristics included in the previously-stored PSP 215. Insome embodiments, to retrieve the previously-stored PSP 215, theelectronic computing device determines a characteristic of acurrent/in-progress test measurement session including at least one of acharacteristic of the user 135, a characteristic of the first playbacksystem 210, and a characteristic of an environment 130 in which the user135 is viewing the first playback system 210. The electronic computingdevice then may identifying the previously-stored PSP 215 from aplurality of previously stored PSPs 215 based on the previously-storedPSP 215 including one or more of the same characteristics as thecharacteristic of the current/in-progress test measurement session.

For example, the electronic computing device may determine that a storedPSP 215 includes information about the same user 135 but that thecurrent playback system 210 and/or the current environment 130 isdifferent than the stored playback system 210 and/or environment 130(e.g., the same user is watching television on a different television ina different room in their house). Despite the characteristics of thestored PSP 215 not exactly matching the current situation, theelectronic computing device may nevertheless use one or media parametersof the stored PSP 215 as a baseline to begin outputting test media 225during the test measurement session. In other words, the electroniccomputing device may output test media 225 (such as a hybrid image) thatis filtered or otherwise altered in accordance with the stored PSP 215rather than outputting test media 225 randomly or according to a genericmodel. In some situations, outputting test media 225 based on the mediaparameters included in the stored PSP 215 may reduce the duration of thetest measurement session and/or to improve measurement accuracy bestsuited for the current situation. For example, if a contrast sensitivityfunction (CSF) is generated by the electronic computing device tospecify personalized sensitivity information, typically several tens ofmeasurements are required to accurately estimate the media parameters ofthe CSF in one test measurement session. However, when the electroniccomputing device begins the test measurement session from a startingpoint that was already measured for the user 135 in a differentenvironment 130 and/or by a different playback system 210 (or foranother common attribute besides having a common user 135), the numberof measurements required to accurately estimate the media parameters inthe current situation may be reduced compared to the typical amount. Inother words, the spatial frequency and contrast of current stimulus forthe CSF measurement in the current test measurement session may beadjusted according to the user responses 220 of previous test media 225and the estimate of CSF from previously stored PSPs 215.

Along similar lines, in some embodiments, when generating a PSP 215 toestimate and optimize QoE, a single PSP 215 can be estimated frommultiple stored PSPs 215 or selected that closely matches otheridentified attributes (e.g., location, demographic, viewing devicemake/model, screen size, etc.). For example, when the electroniccomputing device detects a change in user 135, environment 130, and/orplayback device 210 and the user 135 elects not to participate in a newtest measurement session, the electronic computing device may generatean estimated PSP 215 based on multiple stored PSPs 215 with a similaruser 135, environment 130, and/or playback device 210.

In use cases where multiple viewers are present (each having a unique orunknown PSP) for a single playback system 210, (e.g., a television at ahome with multiple users/viewers), the electronic computing device mayselect a single PSP 215 based on many different criteria. For example,if the goal of the system 600 is to minimize the risk of perceived QoEdegradation to any users, the electronic computing device may select themost sensitive PSP 215 from among the group of PSPs 215 corresponding toeach of the multiple viewers. In this example, the electronic computingdevice attempts to ensure that even the most sensitive user viewing thedisplay 730 does not experience a decrease QoE. Assuming that the mostsensitive user does not experience a decrease in QoE, it follows thatless sensitive users viewing the same display 730 would also notexperience a decrease in QoE because they are less sensitive to changesin image/video quality than the most sensitive user. In someembodiments, the system 600 may reduce the number of PSP candidates fora given playback system 210 (e.g., a television at a home with multipleusers/viewers) based on user presence information, for example, fetchedfrom other applications (e.g., smart home applications) or GPSinformation of personal mobile devices.

At block 925, the electronic computing device determines whether it hasgathered enough information to complete a personalized sensitivityprofile (PSP) 215. As explained above, this information may be gatheredfrom current user responses 220 to current test media 225 (at block 920)and/or may be retrieved from previously stored PSPs 215 (at block 910).In FIG. 9 , block 910 is shown in dashed lines to indicate that block910 is optional and may not be performed in some implementations of themethod 900. In other words, in some situations, the electronic computingdevice may generate the PSP 215 (at block 930) based on the receiveduser responses 220 to test media 225 without retrieving previouslystored PSPs 215.

Conversely, although blocks 915, 920, and 925 are not shown in dashedlines in FIG. 9 , in some situations, blocks 915, 920, and 925 may notbe performed by the electronic computing device. In other words, theelectronic computing device may not engage in a test measurement sessionin some situations and may instead rely solely on one or more storedPSPs 215 to generate the PSP 215 used for the current media session. Forexample, upon retrieving stored PSPs 215 of the user 135 and or theenvironment 130 (at block 910), the electronic computing device maydetermine that one of the stored PSPs 215 corresponds to the user 135,the environment 130, and the playback device 210. Accordingly, at block930, the electronic computing device may utilize the correspondingpreviously-stored PSP 215 as the PSP 215 for the current media sessionof the user 135 on the playback device 210 in the environment 130. Inthis situation, there is no need for the electronic computing device toengage in a test measurement session because the viewing characteristicsof the current media delivery session were previously stored in a PSP215 during a previous test measurement session.

As another example of the electronic computing device not engaging in atest measurement session (i.e., not performing blocks 915, 920, and 925of FIG. 9 ), as explained above, when the electronic computing devicedetects a change in user 135, environment 130, and/or playback device210 and the user 135 elects not to participate in a new test measurementsession, the electronic computing device may generate an estimated PSP215 based on multiple stored PSPs 215 with a similar user 135,environment 130, and/or playback device 210. For example, if a storedPSP 215 is associated with the same user, the electronic computingdevice may adjust one or more characteristics of the stored PSP 215based on a known change in display size or other display characteristicsbetween the playback system 210 associated with the stored PSP 215 andthe playback system 210 currently being used by the user 135. Similarly,the electronic computing device may adjust one or more characteristicsof the stored PSP 215 based on a known change in the environment of theuser 135. For example, based on sensor data from environmental sensors305, the electronic computing device may determine that the currentenvironment 130 is darker than the environment 130 associated with thestored PSP 215. In other embodiments, instead of generating an estimatedPSP 215, the electronic computing device may retrieve and use a storedPSP 215 that includes similar characteristics as the determined and/orknown characteristics of the user 135, the environment 130, and/or theplayback device 210. For example, the electronic computing device mayretrieve a stored PSP 215 of the user 135 even though the stored PSP 215is for a different environment 130 and/or for a different playbacksystem 210.

Returning back to the explanation of block 925, when the electroniccomputing device determines that more information is desired to completethe PSP 215 (e.g., to more accurately complete a CSF as shown in FIGS.5, 11A, and 11B), the method 900 proceeds back to block 915 to continueoutputting test media 225 and receiving user inputs (i.e., userresponses 220) in response to the test media 225. When the electroniccomputing device determines that it has gathered enough information tocomplete the PSP 215, the method proceeds to block 930.

At block 930, the electronic computing device generates a personalizedsensitivity profile (PSP) of one or more viewing characteristics of theuser based on the user input. For example, the electronic computingdevice generates a personalized CSF 520 that is translated and/or scaledfrom an ideal CSF 515 as shown in FIG. 5 . Additionally oralternatively, algorithms used to determine streaming parameters, suchas the ABR request 140 (or a request with respect to another mediadelivery method), may have coefficients adjusted and stored in the PSP215. In some embodiments, the electronic computing device generates apersonalized ABR ladder (or another personalized media delivery method)to be included in the PSP 215.

At block 935, the electronic computing device determines, based at leastin part on the PSP 215, a media parameter. For example, the electroniccomputing device determines a value of a media parameter (e.g., a valueof one or more of a segment size, a bit rate, a resolution, a framerate, another media parameter that affects operation of a videoencoder/transcoder/transrater associated with the media server 105and/or the network 115, etc.) At block 940, the media server 105provides, over the network 115, output media to the playback system 210in accordance with the media parameter. The output media is configuredto be output with the playback system 210 (e.g., an image/videoconfigured to be output on the display 730 of the playback system 210).

To determine the media parameter (i.e., a value of the media parameter)at block 935, the electronic computing device may perform a transform ofa generic objective ABR logic model into a personalized objective ABRlogic model (POM 405) as shown in FIG. 4 and described previously hereinwith respect to FIG. 4 . In some embodiments, the POM 405 may beimplemented within a streaming system supporting adaptive bit ratedelivery such as a television, a Set-top-box, a Digital Media Adapter,or a mobile device as shown in FIG. 6 . In this example, the POM 405 isutilized by the ABR request logic 140 of the playback system 210 toimprove the selection of encoded video and/or audio segments based on,but not limited to, segment size, bit rate, resolution, frame rate,codec, etc. to match, or provide the closest match among the availableencoded segments to a PSP 215 of the user 135. For example, algorithmsthat define the generic objective ABR logic model (or another genericmedia delivery method) that are used to determine streaming parameters,such as the ABR request 140 (or a request with respect to another mediadelivery method), may have their coefficients adjusted/personalized inaccordance with information stored in the PSP 215.

As described previously herein, the ABR Ladder 137 in FIG. 6 representsa collection of available audio and video segments across a range of bitrates, resolutions, frame rates, etc. Also as explained previouslyherein, existing media delivery systems/methods (whether ABR-enabled orotherwise) are inefficient and often lead to the playback system 210requesting more data than it needs for seamless playback and/orrequesting values of media parameters (e.g., a resolution/bit rate/framerate combination) that exceed a PSP 215 of the user 135. In other words,existing media delivery logic attempting to increase the deliveredresolution/bit rate/frame rate beyond a sensitivity threshold(s) of theuser 135 will not translate to increased QoE for the user 135. Thedisclosed POM-based media delivery method translates to more efficientdelivery and therefore a reduction in delivery costs for an over-the-top(OTT) service as these services pay per Gigabyte egressed from theircontent delivery network (CDN) vendors.

In some embodiments, at block 935, the electronic computing deviceselects values of one or more media parameters (e.g., a resolution/bitrate/frame rate combination) that results in streamed media that iswithin the range of sensitivity perception of the user 135. For example,the electronic computing device may use the CSF 1125 of FIG. 11A tocontrol ABR requests 140 to the media server 105 to request a resolutionof streamed media at approximately 720p because the CSF 1125 indicatesthat the sensitivity peak 1110 of a first user is at approximately 720p.On the other hand, the electronic computing device may use the CSF 1170of FIG. 11B to control ABR requests 140 to the media server 105 torequest a resolution of streamed media at a lower resolution ofapproximately 540p because the CSF 1170 indicates that the sensitivitypeak 1175 of a second user is at approximately 540p. In the twoimmediately above examples, a first media parameter (i.e., 540presolution) and a second media parameter (i.e., 720p resolution) aredetermined such that providing of first output media to a first playbackdevice 210 of the second user decreases a first usage of resources ofthe network 115 to be lower than a second usage of resources of thenetwork 115 with respect to providing of second output media to a secondplayback device 210 of the first user. Nevertheless, despite thisdifference in usage of resources when streaming output media to the twoplayback devices 210, a first percentage of a first personalized QoE ofthe first user is maintained at approximately the same level as a secondpercentage of a second personalized QoE of the second user (see FIG. 12Band Table 1B).

Although the method 900 is described above with respect to a mediasession of a single playback system 210 or of two playback systems 210,in some embodiments, the method 900 may be performed with respect toadditional playback systems 210. For example, the method 900 may be usedto determine a PSP 215 for each of a plurality of playback systems 210that are receiving media streams from a particular node on the network115. The electronic computing device may improve/optimize one or moremedia parameters (e.g., coding and delivery parameters) of each mediastream being provided to each of the plurality of playback systems 210to improve/optimize the media streams in aggregate/as a whole from thenetwork 115.

For example, for mobile wireless and broadband network operators, thedisclosed POM-based media delivery and coding method 900 can beleveraged to add additional capacity to existing access networks withouttrading off end user/viewer QoE. In some embodiments, the method 900provides network operators with a new method to reduce the rate ofcapital investment necessary to increase network capacity. FIGS. 12A and12B and tables 1A and 1B illustrate example bandwidth and QoE statisticsof the network 115 when using an existing streaming method to streammedia versus using the method 900 to stream media. A server using anexisting method is referred to as an existing server while a serverusing the method 900 is referred to as an xCD server (i.e., anExperience Coding and Delivery server).

FIG. 12A illustrates a chart 1205 for a network with an uncappedbandwidth. The upper curve 1210 represents a bandwidth used by mediastreamed by an existing server. The lower curve 1215 represents abandwidth used by media streamed by an xCD server using the method 900.As shown in FIG. 12A, the bandwidth used by the media streamed by thexCD server using the method 900 is approximately 20% less than thebandwidth used by the same media streamed by the existing server.Additionally, as indicated by below Table 1A that corresponds to FIG.12A, the QoE of all viewers (e.g., both high sensitivity users [i.e.,close viewers] and low sensitivity users [i.e., far viewers]) remains at100% (i.e., perfect QoE).

TABLE 1A QoE - Existing Server QoE xCD Server Ten closest Ten farthestTen closest Ten farthest users users users users 100% 100% 100% 100%

As indicated by FIG. 12A and Table 1A, the method 900 results in moreefficient media delivery without reducing user QoE for a population ofusers that each receive a unicast session based on their respective PSP215. This increase in efficiency and decrease in bandwidth withoutreducing user QoE is a result of the system 600 reducing the bit rate,resolution, etc. of streamed media based on a PSP 215 of a user 135 whodoes not experience increased QoE when the bit rate, resolution, etc.increases beyond a certain point. In other words, in some embodiments,the method 900 may aim to deliver the highest perceptible quality ofmedia to each user without delivering higher quality media to anyspecific user than can be perceived by the specific user (i.e.,personalized media content delivery).

FIG. 12B illustrates a chart 1250 for a network with a cappedbandwidth/fixed network capacity (e.g., approximately 60 Mbps). Thecurve 1255 represents a bandwidth used by media streamed by an existingserver. The curve 1260 represents a bandwidth used by media streamed byan xCD server using the method 900. Unlike the curves of FIG. 12A, thecurves 1255 and 1260 of FIG. 12B use approximately the same bandwidthover time. In some embodiments, the bandwidth used by the media streamedby the xCD server using the method 900 may be approximately 1% less thanthe bandwidth used by the same media streamed by the existing server.However, as indicated by below Table 1B that corresponds to FIG. 12B,use of the method 900 by the xCD server over a fixed-capacity networklink yields a more uniform reduction in QoE across high and lowsensitivity users compared to the existing server that implements atraditional ABR segment selection approach.

TABLE 1B QoE - Existing Server QoE xCD Server Ten closest Ten farthestTen closest Ten farthest users users users users 60.7% 82.7% 80.2% 78.4%

For example, Table 1B indicates that high sensitivity users (i.e., theclosest users to their respective playback systems 210) experienceapproximately a 40% reduction in QoE in the capped bandwidth networkwhen media is streamed by the existing server. Comparatively, the lowsensitivity users (i.e., the farthest users from their respectiveplayback systems 210) experience only approximately a 20% reduction inQoE in the capped bandwidth network when media is streamed by theexisting server. This difference in reduction of QoE is caused by theexisting server reducing streaming quality for all users in an equalmanner even though changes in streaming quality affect different usersdifferently.

On the other hand, because the user PSPs 215 are used by the xCD serverto more intelligently reduce streaming quality in a different manner fordifferent users, the same capped bandwidth network is able to providemore uniform reduction in QoE between all users of the system 600. Insome embodiments, the more uniform reduction in QoE results in a higheroverall QoE for the users of the system 600. For example, Table 1Bindicates that high sensitivity users (i.e., the closest users to theirrespective playback systems 210) experience only approximately a 20%reduction in QoE in the capped bandwidth network when media is streamedby the xCD server. Similarly, the low sensitivity users (i.e., thefarthest users from their respective playback systems 210) experienceonly approximately a 20% reduction in QoE in the capped bandwidthnetwork when media is streamed by the xCD server. In other words, asindicated by Table 1B, the xCD server implementing the method 900 maysignificantly improve the QoE of the high sensitivity users while onlymoderately reducing or maintaining the QoE of the low sensitivity users.

FIGS. 13A and 13B illustrate another example of how the method 900 mayallow more users/subscribers to stream media on a fixed-capacity networkwithout negatively impacting QoE. FIGS. 13A and 13B show example graphs1305 and 1350 of a number of subscribers served (x-axis) versus apercentage of subscribers experiencing reduced QoE (y-axis) using a 1.89Gbit/s (4096-QAM) fixed-capacity network with an 88% effectivethroughput after accounting for overhead. The graphs 1305, 1350 of FIGS.13A and 13B assume a 50% split between viewers with high sensitivity(e.g., three picture heights [3H] away from their respective playbacksystem 210) and low sensitivity (e.g., six picture heights [6H] awayfrom their respective playback system 210).

The graphs 1305, 1350 demonstrate that as users/subscribers are added tothe fixed capacity network, video resolution (as an example) must bedowngraded once the number of users/subscribers reaches a certainthreshold 1310, 1355. However, similar to the above example with respectto FIGS. 12A and 12B and Tables 1A and 1B, downgrading the resolution ofall users/subscribers equally (as shown in FIG. 13A) results in unequalQoE depending on whether users/subscribers are watching at 3H or at 6H.For example, the users watching at 3H generally perceive a largerdecrease in QoE than the users watching at 6H when the same reduction inresolution is implemented on all media streams. This difference in QoEreduction between different types of users is illustrated by the 3Hcurve 1315 and the 6H curve 1320 shown in FIGS. 13A and 13B. A shadedarea 1325 between the 3H curve 1315 and the 6H curve 1320 illustratesthe inequality of the reduced QoE (i.e., perception of degraded service)between high sensitivity users/subscribers and low sensitivityusers/subscribers. For example, for the network to serve 1200 users, thereduction in video quality is visible to 60% of users at 3H (i.e., highsensitivity users), but only to about 2% of users at 6H (i.e., lowsensitivity users).

FIG. 13A illustrates a first QoE curve 1330 that indicates the QoEexperienced by different users/subscribers when utilizing a streamingmanagement method that adapts video coded bit rate and resolution basedon viewing distance of all users/subscribers in the same way. On theother hand, FIG. 13B illustrates a second QoE curve 1360 that indicatesthe QoE experienced by different users/subscribers when the method 900of FIG. 9 is used to personalize the downgrade in resolution todifferent users as additional users are added to the network. Asexplained above, FIGS. 13A and 13B both assume a 50% split between 3Hand 6H viewing distance among the users/subscribers.

Using the method 900, the electronic computing device controlling mediaparameters understands which users are watching at what distance and howeach user's QoE will be affected by a reduction in resolution (e.g.,based on information stored in the PSP 215 of each user). Accordingly,the electronic computing device executing the method 900 can allocate abit rate/resolution combination so as to achieve equal average QoE inboth groups of users (i.e., high sensitivity users and low sensitivityusers). This improvement is illustrated by the difference between theQoE curve 1330 in FIG. 13A and the QoE curve 1360 in FIG. 13B. Forexample, the threshold 1310 of FIG. 13A at which any user experiencesdecreased QoE is when the network is serving approximately 600 users.Comparatively, the threshold 1355 of FIG. 13B is approximately doubledto about 1200 users. In other words, the network executing the method900 can serve approximately double the users that an existing networkcan serve without any users experiencing decreased QoE.

Returning to FIG. 9 , as indicated by the dashed line arrow, in someembodiments, the electronic computing device repeats blocks 935 and 940.Repeating of these blocks may allow for real-time tracking of sensoroutputs in the playback environment 130 (e.g., environment sensors 305)and/or the network 115 during media delivery services and correspondingadjusting of the personalized objective model 405 in dynamic/adaptivemanners for versatile use cases. For example, the system 600 maydetermine that a room in which the playback system 210 is located hasgotten darker since the user 135 began watching television (e.g., due tothe sun setting, due to the user 135 closing shades in the room, etc.).In response to this determination, the system 600 may adjust thepersonalized objective model 405 by, for example, selecting a differentstored PSP 215 that includes an ambient light characteristic that ismore similar to that of the now darker room.

As explained previously herein, the ABR ladder 137 and ABR-selectionmethods referred to herein are merely one example method that can beused by the system 600 to control media delivery from the media server105 to the playback system 210 over the network 115. In otherembodiments, other methods may be used to dynamically adjust videoencoder/transcoder/transrater parameters (i.e., media parameters) suchas bit rate and/or resolution of encoded media that is being streamed.Similar to the ABR-related methods included in many examples, theseother media delivery methods have their media parameters adjusted basedon one or more PSPs 215 to optimize media delivery as described herein.In some embodiments, the media delivery methods are upstream mediadelivery methods implemented by the media server 105 and/or the network115 (i.e., upstream of the playback system 210).

It is to be understood that the embodiments are not limited in itsapplication to the details of the configuration and arrangement ofcomponents set forth herein or illustrated in the accompanying drawings.The embodiments are capable of being practiced or of being carried outin various ways. Also, it is to be understood that the phraseology andterminology used herein are for the purpose of description and shouldnot be regarded as limiting. The use of “including,” “comprising,” or“having” and variations thereof are meant to encompass the items listedthereafter and equivalents thereof as well as additional items. Unlessspecified or limited otherwise, the terms “mounted,” “connected,”“supported,” and “coupled” and variations thereof are used broadly andencompass both direct and indirect mountings, connections, supports, andcouplings.

In addition, it should be understood that embodiments may includehardware, software, and electronic components or modules that, forpurposes of discussion, may be illustrated and described as if themajority of the components were implemented solely in hardware. However,one of ordinary skill in the art, and based on a reading of thisdetailed description, would recognize that, in at least one embodiment,the electronic-based aspects may be implemented in software (e.g.,stored on non-transitory computer-readable medium) executable by one ormore electronic processors, such as a microprocessor and/or applicationspecific integrated circuits (“ASICs”). As such, it should be noted thata plurality of hardware and software based devices, as well as aplurality of different structural components, may be utilized toimplement the embodiments. For example, “servers” and “computingdevices” described in the specification can include one or moreelectronic processors, one or more computer-readable medium modules, oneor more input/output interfaces, and various connections (e.g., a systembus) connecting the various components.

Various features and advantages are set forth in the following claims.

What is claimed is:
 1. A method for delivering media to a playbackdevice, the method comprising: outputting, with a first playback deviceand during a first test measurement session, first test media to beviewed by a first user; receiving a first user input from the firstuser, the first user input related to a first perception of the firsttest media by the first user and indicating a first personalized qualityof experience of the first user with respect to the first test media;generating, with one or more electronic processors, a first personalizedsensitivity profile including one or more viewing characteristics of thefirst user based on the first user input; determining, with the one ormore electronic processors and based at least in part on the firstpersonalized sensitivity profile, a first media parameter, the firstmedia parameter being determined in order to increase an efficiency ofmedia delivery to the first playback device over a network whilepreserving the first personalized quality of experience of the firstuser; and providing, over the network, first output media to the firstplayback device in accordance with the first media parameter, the firstoutput media configured to be output with the first playback device. 2.The method of claim 1, wherein providing the first output media to thefirst playback device in accordance with the first media parameterresults in the network using decreased bandwidth to provide the firstoutput media to the first playback device without reducing the firstpersonalized quality of experience of the first user.
 3. The method ofclaim 1 or claim 2, further comprising: outputting, with a secondplayback device and during a second test measurement session, secondtest media to be viewed by a second user; receiving a second user inputfrom the second user, the second user input related to a secondperception of the second test media by the second user and indicating asecond personalized quality of experience of the second user withrespect to the second test media, wherein the second personalizedquality of experience indicates that the second user is more sensitiveto decreases in quality of media than the first user of the firstplayback device; generating, with the one or more electronic processors,a second personalized sensitivity profile of one or more viewingcharacteristics of the second user based on the second user input;determining, with the one or more electronic processors, a second mediaparameter for the second playback device based at least in part on thesecond personalized sensitivity profile; providing, with the one or moreelectronic processor and over the network, second output media to thesecond playback device in accordance with the second media parameter,the second output media configured to be output with the second playbackdevice; wherein the first media parameter and the second media parameterare determined such that providing of the first output media to thefirst playback device decreases a first usage of resources of thenetwork to be lower than a second usage of resources of the network withrespect to providing of the second output media to the second playbackdevice; and wherein a first percentage of the first personalized qualityof experience of the first user is maintained at approximately the samelevel as a second percentage of the second personalized quality ofexperience of the second user.
 4. The method of claim 3, wherein the oneor more electronic processors includes at least one of a firstelectronic processor of the first playback device, a second electronicprocessor of the second playback device, and a third electronicprocessor associated with the network or a media server.
 5. The methodof any one of claims 1 to 4, wherein the first test media includes aplurality of hybrid images and the first user input includes a pluralityof user inputs, each of the first user inputs being received in responseto a respective hybrid image of the plurality of hybrid images, andfurther comprising: generating, with the one or more electronicprocessors, each hybrid image by low-pass filtering a first source imageto create a low-pass filtered source image, high-pass filtering a secondsource image to create a high-pass filtered source image, and overlayingthe low-pass filtered source image and the high-pass filtered sourceimage on top of each other to create the hybrid image; wherein eitherthe low-pass filtered source image or the high-pass filtered sourceimage is perceived by the first user as dominant depending on the one ormore viewing characteristics of the first user.
 6. The method of claim5, wherein outputting the first test media includes: outputting, withthe first playback device, a first hybrid image to be viewed by thefirst user; receiving the first user input from the first user, thefirst user input related to the first perception of the first hybridimage by the first user; and outputting, with the first playback device,a second hybrid image to be viewed by the first user, wherein a cutofffrequency of at least one of the low-pass filtering and the high-passfiltering used to create the second hybrid image is based on the firstuser input related to the first perception of the first hybrid image bythe first user.
 7. The method of any one of claims 1 to 6, whereinoutputting the first test media includes: retrieving, with the one ormore electronic processors and from a memory, a previously-storedpersonalized sensitivity profile; and generating, with the one or moreelectronic processors, the first test media based on one or more viewingcharacteristics included in the previously-stored personalizedsensitivity profile.
 8. The method of claim 7, wherein retrieving thepreviously-stored personalized sensitivity profile includes:determining, with the one or more electronic processors, acharacteristic of the first test measurement session including at leastone of a characteristic of the first user, a characteristic of the firstplayback device, and a characteristic of an environment in which thefirst user is viewing the first playback device; and identifying, withthe one or more electronic processors, the previously-storedpersonalized sensitivity profile from a plurality of previously storedpersonalized sensitivity profiles based on the previously-storedpersonalized sensitivity profile including one or more of the samecharacteristics as the characteristic of the first test measurementsession.
 9. The method of any one of claims 1 to 8, wherein determiningthe first media parameter includes: retrieving, with the one or moreelectronic processors and from a memory, a generic objective modelconfigured to control media streaming based on at least one of resourceavailability of the network and playback system parameters;transforming, with the one or more electronic processors, the genericobjective model into a personalized objective model using the firstpersonalized sensitivity profile; and providing, over the network, thefirst output media to the first playback device in accordance with thepersonalized objective model.
 10. The method of claim 9, whereintransforming the generic objective model into the personalized objectivemodel includes at least one of translating and scaling an ideal contrastsensitivity function (CSF) used by the generic objective model to createa personalized CSF based on the first personalized sensitivity profile.11. The method of any one of claims 1 to 10, wherein the one or moreviewing characteristics of the first user include at least one of aviewing distance between the first user and a display of the firstplayback device, lighting in an environment in which the first user isviewing the display, and vision sensitivity of eyes of the first user.12. The method of any one of claims 1 to 11, further comprising:determining, with an environmental sensor in an environment where theplayback device is located, an environmental condition; whereingenerating the first personalized sensitivity profile including the oneor more viewing characteristics of the first user includes generating,with the one or more electronic processors, the first personalizedsensitivity profile such that first sensitivity profile includes theenvironmental condition.
 13. An electronic computing device comprising:a first playback device including a display, wherein the display isconfigured to output media to a first user; and one or more electronicprocessors communicatively coupled to the display, the one or moreelectronic processors configured to output, with the first playbackdevice and during a first test measurement session, first test media tobe viewed by the first user, receive a first user input from the firstuser, wherein the first user input is related to a first perception ofthe first test media by the first user and indicates a firstpersonalized quality of experience of the first user with respect to thefirst test media, generate a first personalized sensitivity profileincluding one or more viewing characteristics of the first user based onthe first user input, determine, based at least in part on the firstpersonalized sensitivity profile, a first media parameter, wherein thefirst media parameter is determined in order to increase an efficiencyof media delivery to the first playback device over a network whilepreserving the first personalized quality of experience of the firstuser, and provide, over the network, first output media to the firstplayback device in accordance with the first media parameter, whereinthe first output media is configured to be output with the firstplayback device.
 14. The electronic computing device of claim 13,wherein the one or more electronic processors include at least one of afirst electronic processor of the first playback device and a secondelectronic processor associated with the network or a media server. 15.A method for displaying a hybrid image on a playback device, the methodcomprising: determining, with one or more electronic processors of anelectronic computing device, a first value of a media parametersupported by a media server and a network configured to stream media;determining, with the one or more electronic processors, a second valueof the media parameter supported by the media server and the network; atleast one of generating and selecting, with the one or more electronicprocessors, the hybrid image based on the first value of the mediaparameter and the second value of the media parameter such that thehybrid image includes a first interpretation corresponding to the firstvalue of the media parameter and a second interpretation correspondingto the second value of the media parameter; and displaying, on a displayof the playback device, the hybrid image.
 16. The method of claim 15,wherein the media parameter includes at least one of a video resolution,a bit rate, and a frame rate of media streaming from the media serverover the network.
 17. The method of claim 15 or claim 16, furthercomprising: receiving, with an input device of the playback device, auser input from a user, the user input indicating that the firstinterpretation is perceived by the user when the hybrid image isdisplayed on the display; determining, with the one or more electronicprocessors, a third value of the media parameter supported by the mediaserver and the network based on the user input; at least one ofgenerating and selecting, with the one or more electronic processors, asecond hybrid image based on the first value of the media parameter andthe third value of the media parameter such that the second hybrid imageincludes a third interpretation corresponding to the third value of themedia parameter and a fourth interpretation corresponding to the firstvalue of the media parameter; and displaying, on the display of theplayback device, the second hybrid image.
 18. The method of any one ofclaims 15 to 17, wherein at least one of generating and selecting thehybrid image includes generating, with the one or more electronicprocessors, the hybrid image by low-pass filtering a first source imageto create a low-pass filtered source image, wherein a cutoff frequencyof the low-pass filtering is based on the first value of the mediaparameter; high-pass filtering a second source image to create ahigh-pass filtered source image, wherein a cutoff frequency of thehigh-pass filtering is based on the second value of the media parameter;and overlaying the low-pass filtered source image and the high-passfiltered source image on top of each other to create the hybrid image;wherein either the low-pass filtered source image or the high-passfiltered source image is perceived by a user as dominant depending onone or more viewing characteristics of the user when the hybrid image isdisplayed on the display.
 19. The method of any one of claims 15 to 18,further comprising: receiving, with an input device of the playbackdevice, a user input from a user, the user input indicating that thefirst interpretation is perceived by the user when the hybrid image isdisplayed on the display; determining, with the one or more electronicprocessors and based on the user input, that the user is more sensitiveto the first value of the media parameter than to the second value ofthe media parameter; generating, with the one or more electronicprocessors, a personalized sensitivity profile of viewingcharacteristics of the user based on the determination that the user ismore sensitive to the first value of the media parameter, thepersonalized sensitivity profile including the first value of the mediaparameter; and providing, with the media server over the network, outputmedia to the playback device in accordance with the personalizedsensitivity profile, the output media configured to be output with theplayback device.
 20. The method of claim 19, further comprising:determining, with the one or more electronic processors and based on theuser input, at least one of a subset of spatial frequencies of thehybrid image to which the user is most sensitive and a sizing of thehybrid image to which the user is most sensitive; wherein the viewingcharacteristics of the personalized sensitivity profile include the atleast one of the subset of spatial frequencies of the hybrid image towhich the user is most sensitive and the sizing of the hybrid image towhich the user is most sensitive.